Project Outline

1. Data Loading and Preprocessing

   1. Load the dataset.
   2. Exploratory Data Analysis:
      • Feature histograms.
      • Train-Test Split (to avoid leakage).
      • Check for outliers (and optional removal).
      • Scaling (Standardization).
      • Correlation Matrix and Multicollinearity Check:
        • Remove highly correlated features.
   3. Prepare final features for analysis.

2. Linear Algorithms

   1. Perceptron
      • Train with 5-fold cross-validation (CV).
      • Hyperparameter tuning using CV.
      • Evaluate the best-tuned model on the test set.
   2. Pegasos SVM
      • Train with 5-fold cross-validation.
      • Evaluate the model on the test set.
   3. Logistic Regression
      • Train with 5-fold cross-validation.
      • Evaluate the model on the test set.
   4. Comparison of Results
      • Summarize and compare accuracies across Perceptron, Pegasos SVM, and Logistic Regression.

3. Polynomial Feature Expansion

   1. Expand features to degree 2 (include squared terms and interactions).
   2. Retrain and evaluate:
      • Perceptron with polynomial expansion.
      • Pegasos SVM with polynomial expansion.
      • Logistic Regression with polynomial expansion.
   3. Summarize and compare results:
      • Linear vs. Polynomial-expanded features.
      • Visualize results.

4. Kernelized Methods

   1. Kernelized Perceptron
      • Implement and evaluate with:
        • Gaussian Kernel.
        • Polynomial Kernel.
      • Compare test set accuracies.
   2. Kernelized Pegasos SVM
      • Implement and evaluate with:
        • Gaussian Kernel.
        • Polynomial Kernel.
      • Compare test set accuracies.

5. Final Comparison

   1. Summarize final results:
      • Linear models.
      • Polynomial-expanded models.
      • Kernelized models.

Kernelized Linear Classification

Initial imports + functions

Basic libraries + custom functions (ffunctions.py):

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Uncomment the following line if you want that plots appear inline in a real Jupyter notebook:
# %matplotlib inline

from ffunctions import (
    load_data,
    train_test_split,
    detect_outliers_zscore,
    remove_outliers,
    standard_scaler_fit,
    standard_scaler_transform,
    check_high_correlation,
    k_fold_cross_validation,
    perceptron_classifier_func,
    pegasos_classifier_func,
    logistic_regression_classifier_func,
    perceptron_classifier_func_eta,
    perceptron_train,
    perceptron_predict,
    pegasos_train,
    pegasos_predict,
    logistic_regression_train,
    logistic_regression_predict,
    polynomial_feature_expansion,
    gaussian_kernel,
    polynomial_kernel,
    kernelized_perceptron_classifier_func,
    kernelized_perceptron_train,
    kernelized_perceptron_predict,
    kernelized_pegasos_classifier_func,
    kernelized_pegasos_train,
    kernelized_pegasos_predict
)

best_hparams_table = {
    "Perceptron": {},
    "Pegasos": {},
    "Logistic": {},
    "PerceptronPoly": {},
    "PegasosPoly": {},
    "LogisticPoly": {},
    "KernelPerceptronGauss": {},
    "KernelPerceptronPoly": {},
    "KernelPegasosGauss": {},
    "KernelPegasosPoly": {}
}

Data loading:

csv_path = "/Users/michelecoaro/Documents/GitHub/KLC/klc_project/dataset.csv"
df = load_data(csv_path)
df.head()

	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10	y
0	1.205492	5.823226	98.837539	-1.075852	0.999205	0.911543	3.623558	-1.720267	-0.346191	-54.708330	-1
1	1.692571	-0.887019	100.901276	-0.595548	0.177550	-0.915495	4.320264	0.907834	3.126815	-56.397484	-1
2	4.289320	1.416843	100.784735	-2.897154	-0.066972	-0.786173	2.093003	1.336237	2.183829	-56.197728	1
3	0.542420	-1.010095	100.015580	-3.070705	0.088324	-0.242669	0.767942	-0.284683	-2.104145	-55.794045	1
4	2.431765	-1.224177	100.709237	-0.102892	0.699031	-0.656315	-0.911784	-1.355873	1.537825	-55.917863	-1

Exploratory Data Analysis

Understanding feature locations and shape

feature_cols = [f"x{i}" for i in range(1, 11)]
df[feature_cols].hist(bins=30, figsize=(15, 10))
plt.suptitle("Histograms of Numerical Features", fontsize=16)
plt.show()

#pairplot of features with different colors for each feature in the pair
import seaborn as sns
sns.pairplot(df, hue='y', diag_kind='hist')
plt.suptitle("Pairplot of Numerical Features", fontsize=16)
plt.show()

Train - Test split as first data leakage countermeasure:

train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)

print("Train set shape:", train_df.shape)
print("Test set shape:", test_df.shape)

Train set shape: (7788, 11)
Test set shape: (1947, 11)

Checking for outliers in training set

outlier_indices = detect_outliers_zscore(train_df, feature_cols, z_thresh=3.0)
print(f"Number of outliers detected in training set: {len(outlier_indices)}")

Number of outliers detected in training set: 82

Outliers removal:

train_df_no_outliers = remove_outliers(train_df, outlier_indices)
print("Train set (no outliers) shape:", train_df_no_outliers.shape)

Train set (no outliers) shape: (7706, 11)

Applying consistent scaling

means, stds = standard_scaler_fit(train_df_no_outliers, feature_cols)
train_scaled = standard_scaler_transform(train_df_no_outliers, feature_cols, means, stds)
test_scaled = standard_scaler_transform(test_df, feature_cols, means, stds)

# histogram plot for numerical variables after scaling
train_scaled[feature_cols].hist(bins=30, figsize=(15, 10))
plt.suptitle("Histograms of Numerical Features (After Scaling)", fontsize=16)
plt.show()

#side by side plot of features before and after scaling

fig, axs = plt.subplots(2, 5, figsize=(20, 8))
fig.suptitle("Side-by-Side Histograms of Numerical Features Before and After Scaling", fontsize=16)

for i, col in enumerate(feature_cols):
    train_df[col].hist(bins=30, ax=axs[i // 5, i % 5])
    axs[i // 5, i % 5].set_title(f"Before Scaling: {col}")

fig, axs = plt.subplots(2, 5, figsize=(20, 8))
for i, col in enumerate(feature_cols):
    train_scaled[col].hist(bins=30, ax=axs[i // 5, i % 5])
    axs[i // 5, i % 5].set_title(f"After Scaling: {col}")

plt.show()

Correlation Matrix

# 5. Correlation Matrix (for numerical columns)
corr_matrix = train_scaled[feature_cols].corr()
plt.figure(figsize=(8, 6))
plt.imshow(corr_matrix, cmap='coolwarm', aspect='auto')
plt.colorbar(label='Correlation Coefficient')
plt.title('Correlation Matrix')
plt.xticks(range(len(corr_matrix.columns)), corr_matrix.columns, rotation=45)
plt.yticks(range(len(corr_matrix.columns)), corr_matrix.columns)
plt.tight_layout()
plt.show()

Multicollinearity Check

high_corr_pairs = check_high_correlation(train_df_no_outliers, feature_cols, corr_threshold=0.9)
if high_corr_pairs:
    print("High correlation pairs detected (above threshold):")
    for pair in high_corr_pairs:
        print(f"  {pair[0]} and {pair[1]} with correlation = {pair[2]:.3f}")
else:
    print("No pairs of features exceed the correlation threshold.")

High correlation pairs detected (above threshold):
  x3 and x6 with correlation = 0.990
  x3 and x10 with correlation = 0.980
  x6 and x10 with correlation = 0.990

Removing unwanted columns

for col in ["x3", "x6", "x10"]:
    corr = train_df_no_outliers[col].corr(train_df_no_outliers["y"])
    print(col, corr)

final_features = [f for f in feature_cols if f not in ["x3", "x6"]]
print("\nDropping x3, x6 from the final feature set...")
print("Final features:", final_features)

x3 0.051673811986996
x6 -0.05343499561215009
x10 -0.05288351228543469

Dropping x3, x6 from the final feature set...
Final features: ['x1', 'x2', 'x4', 'x5', 'x7', 'x8', 'x9', 'x10']

Algorithms Implementation

Cross-validation

target_col = "y"

# Example: Perceptron with 5 epochs
accuracy_perc = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func(
        tr_fold, vl_fold, f, t, epochs=5
    )
)
print(f"Perceptron 5-fold CV Accuracy: {accuracy_perc:.3f}")

# Example: Pegasos
accuracy_pegasos = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: pegasos_classifier_func(
        tr_fold, vl_fold, f, t, lambda_param=0.01, epochs=5
    )
)
print(f"Pegasos 5-fold CV Accuracy: {accuracy_pegasos:.3f}")

# Example: Logistic Regression
accuracy_logreg = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: logistic_regression_classifier_func(
        tr_fold, vl_fold, f, t, lambda_param=0.01, epochs=5, eta=1.0
    )
)
print(f"Logistic Regression 5-fold CV Accuracy: {accuracy_logreg:.3f}")

Perceptron 5-fold CV Accuracy: 0.585
Pegasos 5-fold CV Accuracy: 0.723
Logistic Regression 5-fold CV Accuracy: 0.718

Perceptron

Hyperparameter Tuning

target_col = "y"
epochs_list = [5, 10, 20, 50]
learning_rates = [0.01, 0.1, 1.0]

tuning_results_perc = {}

for ep in epochs_list:
    for eta in learning_rates:
        acc = k_fold_cross_validation(
            df=train_scaled,
            features=final_features,
            target=target_col,
            k=5,
            random_state=42,
            classifier_func=lambda tr, vl, f, t: perceptron_classifier_func_eta(
                tr, vl, f, t, epochs=ep, eta=eta
            )
        )
        tuning_results_perc[(ep, eta)] = acc

best_params_perc = max(tuning_results_perc, key=tuning_results_perc.get)
best_epochs_perc, best_eta_perc = best_params_perc
best_acc_perc = tuning_results_perc[best_params_perc]

print("Perceptron best hyperparams =>", best_params_perc, "CV Accuracy =", best_acc_perc)

# Save in best_hparams_table
best_hparams_table["Perceptron"]["epochs"] = best_epochs_perc
best_hparams_table["Perceptron"]["eta"] = best_eta_perc
best_hparams_table["Perceptron"]["cv_accuracy"] = best_acc_perc

# Train final model on entire train set with best hyperparams
from ffunctions import perceptron_train_eta, perceptron_predict

theta_perc, theta0_perc = perceptron_train_eta(
    train_scaled, final_features, target_col,
    epochs=best_epochs_perc, eta=best_eta_perc
)
test_preds_perc = perceptron_predict(test_scaled, final_features, theta_perc, theta0_perc)
test_acc_perc = np.mean(test_preds_perc == test_scaled[target_col])
print(f"Perceptron final test accuracy (best hyperparams): {test_acc_perc:.3f}")

Perceptron best hyperparams => (10, 0.1) CV Accuracy = 0.6466382349797284
Perceptron final test accuracy (best hyperparams): 0.601

# Compare default and tuned Perceptron across feature subsets
subsets = {
    "Features": feature_cols}

comparison_results = {}

for subset_name, subset_features in subsets.items():
    print(f"\nEvaluating subset: {subset_name}")
    
    # Default Perceptron
    accuracy_default_perc = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func(
            tr_fold, vl_fold, f, t, epochs=5  # Default epochs
        )
    )
    
    # Tuned Perceptron
    accuracy_tuned_perc = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func_eta(
            tr_fold, vl_fold, f, t, epochs=best_epochs_perc, eta=best_eta_perc
        )
    )
    
    comparison_results[subset_name] = {
        "Default Perceptron": accuracy_default_perc,
        "Tuned Perceptron": accuracy_tuned_perc,
    }
    
    print(f"  Default Perceptron Accuracy: {accuracy_default_perc:.3f}")
    print(f"  Tuned Perceptron Accuracy: {accuracy_tuned_perc:.3f}")

Evaluating subset: Features
  Default Perceptron Accuracy: 0.624
  Tuned Perceptron Accuracy: 0.627

Prediction on test dataset

theta_perc, theta0_perc = perceptron_train(
    train_scaled,
    features=final_features,
    target=target_col,
    epochs=5
)
test_preds_perc = perceptron_predict(
    test_scaled,
    features=final_features,
    theta=theta_perc,
    theta_0=theta0_perc
)

test_acc_perc = np.mean(test_preds_perc == test_scaled[target_col])
print(f"Linear Perceptron final test accuracy: {test_acc_perc:.3f}")

Linear Perceptron final test accuracy: 0.670

Pegasos SVM

Hyperparameter Tuning

lambda_values_pega = [0.001, 0.01, 0.1]
epochs_values_pega = [5, 10, 20]

tuning_results_pegasos = {}
for lam in lambda_values_pega:
    for e in epochs_values_pega:
        acc = k_fold_cross_validation(
            df=train_scaled,
            features=final_features,
            target=target_col,
            k=5,
            random_state=42,
            classifier_func=lambda tr, vl, f, t: pegasos_classifier_func(
                tr, vl, f, t, lambda_param=lam, epochs=e
            )
        )
        tuning_results_pegasos[(lam, e)] = acc

best_lambda_pega, best_epochs_pega = max(tuning_results_pegasos, key=tuning_results_pegasos.get)
best_acc_pega_cv = tuning_results_pegasos[(best_lambda_pega, best_epochs_pega)]

print("Pegasos best hyperparams =>", (best_lambda_pega, best_epochs_pega), "CV =", best_acc_pega_cv)

best_hparams_table["Pegasos"]["lambda"] = best_lambda_pega
best_hparams_table["Pegasos"]["epochs"] = best_epochs_pega
best_hparams_table["Pegasos"]["cv_accuracy"] = best_acc_pega_cv

# Train final Pegasos
theta_pega, theta0_pega = pegasos_train(
    train_scaled, final_features, target_col,
    lambda_param=best_lambda_pega, epochs=best_epochs_pega
)
test_preds_pega = pegasos_predict(test_scaled, final_features, theta_pega, theta0_pega)
test_acc_pega = np.mean(test_preds_pega == test_scaled[target_col])
print(f"Pegasos final test accuracy (best hyperparams): {test_acc_pega:.3f}")

Pegasos best hyperparams => (0.001, 20) CV = 0.7245005727579326
Pegasos final test accuracy (best hyperparams): 0.741

Comparison of different subsets

# Use the best hyperparameters found during tuning
lambda_param_pega = best_lambda_pega
epochs_pega = best_epochs_pega
# Compare Pegasos SVM across feature subsets
pegasos_results = {}

for subset_name, subset_features in subsets.items():
    print(f"\nEvaluating Pegasos SVM on subset: {subset_name}")
    
    accuracy_pegasos = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: pegasos_classifier_func(
            tr_fold, vl_fold, f, t, lambda_param=lambda_param_pega, epochs=epochs_pega
        )
    )
    
    pegasos_results[subset_name] = accuracy_pegasos
    print(f"  Pegasos SVM Accuracy: {accuracy_pegasos:.3f}")

Evaluating Pegasos SVM on subset: Features
  Pegasos SVM Accuracy: 0.711

Measring accuracy of prediction

By testing on test set

theta_pega, theta0_pega = pegasos_train(
    train_scaled,
    features=final_features,
    target=target_col,
    epochs=5
)
test_preds_pega = pegasos_predict(
    test_scaled,
    features=final_features,
    theta=theta_pega,
    theta_0=theta0_pega
)
test_acc_pega = np.mean(test_preds_pega == test_scaled[target_col])
print(f"Pegasos final test accuracy: {test_acc_pega:.3f}")

Pegasos final test accuracy: 0.733

Logistic Regression

Hyperparameter tuning

lambda_list_logreg = [0.001, 0.01, 0.1]
epochs_list_logreg = [5, 10]
eta_list_logreg = [0.01, 0.1, 1.0]

tuning_results_logreg = {}
for lam in lambda_list_logreg:
    for ep in epochs_list_logreg:
        for eta in eta_list_logreg:
            acc = k_fold_cross_validation(
                df=train_scaled,
                features=final_features,
                target=target_col,
                k=5,
                random_state=42,
                classifier_func=lambda tr, vl, f, t: logistic_regression_classifier_func(
                    tr, vl, f, t,
                    lambda_param=lam,
                    epochs=ep,
                    eta=eta
                )
            )
            tuning_results_logreg[(lam, ep, eta)] = acc

best_lam_logreg, best_ep_logreg, best_eta_logreg = max(tuning_results_logreg, key=tuning_results_logreg.get)
best_acc_logreg_cv = tuning_results_logreg[(best_lam_logreg, best_ep_logreg, best_eta_logreg)]

print("Logistic best hyperparams =>", (best_lam_logreg, best_ep_logreg, best_eta_logreg), "CV =", best_acc_logreg_cv)

best_hparams_table["Logistic"]["lambda"] = best_lam_logreg
best_hparams_table["Logistic"]["epochs"] = best_ep_logreg
best_hparams_table["Logistic"]["eta"] = best_eta_logreg
best_hparams_table["Logistic"]["cv_accuracy"] = best_acc_logreg_cv

# Train final logistic
theta_lr, theta0_lr = logistic_regression_train(
    train_scaled, final_features, target_col,
    lambda_param=best_lam_logreg,
    epochs=best_ep_logreg,
    eta=best_eta_logreg
)
test_preds_lr = logistic_regression_predict(test_scaled, final_features, theta_lr, theta0_lr)
test_acc_lr = np.mean(test_preds_lr == test_scaled[target_col])
print(f"Logistic final test accuracy (best hyperparams): {test_acc_lr:.3f}")

Logistic best hyperparams => (0.1, 10, 1.0) CV = 0.7173635291652042
Logistic final test accuracy (best hyperparams): 0.729

# Define parameters for Logistic Regression
lambda_param_logreg = best_lam_logreg
epochs_logreg = best_ep_logreg
eta_logreg = best_eta_logreg

# Compare Logistic Regression across feature subsets
logistic_results = {}

for subset_name, subset_features in subsets.items():
    print(f"\nEvaluating Logistic Regression on subset: {subset_name}")
    
    accuracy_logistic = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: logistic_regression_classifier_func(
            tr_fold, vl_fold, f, t, lambda_param=best_lam_logreg, epochs=best_ep_logreg, eta=best_eta_logreg
        )
    )
    logistic_results[subset_name] = accuracy_logistic
    print(f"  Logistic Regression Accuracy: {accuracy_logistic:.3f}")

Evaluating Logistic Regression on subset: Features
  Logistic Regression Accuracy: 0.714

# Train the Logistic Regression model
theta_lr, theta0_lr = logistic_regression_train(
    train_scaled,
    features=final_features,
    target=target_col,
    lambda_param=best_lam_logreg,  # Regularization parameter
    epochs=best_ep_logreg,
    eta=best_eta_logreg             # Learning rate
)

# Predict on the test set
test_preds_lr = logistic_regression_predict(
    test_scaled,
    features=final_features,
    theta=theta_lr,
    theta_0=theta0_lr
)

# Compute accuracy
test_acc_lr = np.mean(test_preds_lr == test_scaled[target_col])
print(f"Logistic Regression final test accuracy: {test_acc_lr:.3f}")

Logistic Regression final test accuracy: 0.734

Comparison

# Combine results into a DataFrame
comparison_with_logistic = pd.DataFrame({
    "Default Perceptron Loss": [comparison_results[name]["Default Perceptron"] for name in subsets.keys()],
    "Tuned Perceptron Accuracy": [comparison_results[name]["Tuned Perceptron"] for name in subsets.keys()],
    "Pegasos SVM Accuracy": [pegasos_results[name] for name in subsets.keys()],
    "Logistic Regression Accuracy": [logistic_results[name] for name in subsets.keys()]
}, index=list(subsets.keys()))

# Display the comparison table
print("\nComparison of Perceptron, Pegasos, and Logistic Regression on training set:")
display(comparison_with_logistic)

# Comparison table on test set
comparison_test = pd.DataFrame({
    "Perceptron (Default)": [test_acc_perc],
    "Perceptron (Tuned)": [comparison_results["Features"]["Tuned Perceptron"]],
    "Pegasos SVM": [test_acc_pega],
    "Logistic Regression": [test_acc_lr]
}, index=["Test Set"])

# Display the comparison table
print("\nComparison of Perceptron, Pegasos, and Logistic Regression on test set:")
display(comparison_test)

Comparison of Perceptron, Pegasos, and Logistic Regression on training set:

	Default Perceptron Loss	Tuned Perceptron Accuracy	Pegasos SVM Accuracy	Logistic Regression Accuracy
Features	0.62392	0.626535	0.711005	0.714379

Comparison of Perceptron, Pegasos, and Logistic Regression on test set:

	Perceptron (Default)	Perceptron (Tuned)	Pegasos SVM	Logistic Regression
Test Set	0.670262	0.626535	0.732922	0.73395

Polynomial feature expansion

# Expand the scaled training data (train_scaled) and test data (test_scaled)
# using the final_features (excluding x3, x6). Then rescale the newly created polynomial columns.

# 1) Expand polynomial features on the *scaled* train and test data.
train_poly_unscaled = polynomial_feature_expansion(train_scaled, final_features, degree=2, include_bias=False)
test_poly_unscaled = polynomial_feature_expansion(test_scaled, final_features, degree=2, include_bias=False)

# Identify new polynomial columns (excluding 'y')
expanded_feature_cols = [col for col in train_poly_unscaled.columns if col != 'y']

print("Polynomial expansion complete.")
print("Number of expanded features:", len(expanded_feature_cols))
print("Sample of expanded features:", expanded_feature_cols[:10])

Polynomial expansion complete.
Number of expanded features: 46
Sample of expanded features: ['x1', 'x1^2', 'x2', 'x2^2', 'x4', 'x4^2', 'x5', 'x5^2', 'x7', 'x7^2']

# 2) Rescale the newly expanded features
means_poly, stds_poly = standard_scaler_fit(train_poly_unscaled, expanded_feature_cols)
train_poly = standard_scaler_transform(train_poly_unscaled, expanded_feature_cols, means_poly, stds_poly)
test_poly = standard_scaler_transform(test_poly_unscaled, expanded_feature_cols, means_poly, stds_poly)

k = 5
target_col = "y"

Perceptron algorithm with polynomial expansion

acc_perc_poly = k_fold_cross_validation(
    df=train_poly,
    features=expanded_feature_cols,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr, vl, f, t: perceptron_classifier_func_eta(
        tr, vl, f, t, epochs=best_epochs_perc, eta=best_eta_perc
    )
)

theta_perc_poly, theta0_perc_poly = perceptron_train_eta(
    train_poly, expanded_feature_cols, target_col,
    epochs=best_epochs_perc, eta=best_eta_perc
)
test_preds_poly_perc = perceptron_predict(
    test_poly, expanded_feature_cols, theta_perc_poly, theta0_perc_poly
)
test_acc_poly_perc = np.mean(test_preds_poly_perc == test_poly[target_col])

print("\nPerceptron + Poly => CV=%.3f, Test=%.3f" % (acc_perc_poly, test_acc_poly_perc))
best_hparams_table["PerceptronPoly"]["epochs"] = best_epochs_perc
best_hparams_table["PerceptronPoly"]["eta"] = best_eta_perc
best_hparams_table["PerceptronPoly"]["cv_accuracy"] = acc_perc_poly

Perceptron + Poly => CV=0.926, Test=0.924

SVM with PE

lambda_list_poly = [0.001, 0.01, 0.1]
epochs_list_poly = [5, 10, 20]
tuning_results_pegasos_poly = {}

for lam in lambda_list_poly:
    for ep in epochs_list_poly:
        acc = k_fold_cross_validation(
            df=train_poly,
            features=expanded_feature_cols,
            target=target_col,
            k=5,
            random_state=42,
            classifier_func=lambda tr, vl, f, t: pegasos_classifier_func(
                tr, vl, f, t, lambda_param=lam, epochs=ep
            )
        )
        tuning_results_pegasos_poly[(lam, ep)] = acc

best_lam_pega_poly, best_ep_pega_poly = max(tuning_results_pegasos_poly, key=tuning_results_pegasos_poly.get)
acc_pega_poly = tuning_results_pegasos_poly[(best_lam_pega_poly, best_ep_pega_poly)]

print("Pegasos + Poly best hyperparams =>", (best_lam_pega_poly, best_ep_pega_poly), "CV =", acc_pega_poly)
best_hparams_table["PegasosPoly"]["lambda"] = best_lam_pega_poly
best_hparams_table["PegasosPoly"]["epochs"] = best_ep_pega_poly
best_hparams_table["PegasosPoly"]["cv_accuracy"] = acc_pega_poly

# Train final Pegasos on polynomial
theta_pega_poly, theta0_pega_poly = pegasos_train(
    train_poly, expanded_feature_cols, target_col,
    lambda_param=best_lam_pega_poly, epochs=best_ep_pega_poly
)
test_preds_pega_poly = pegasos_predict(
    test_poly, expanded_feature_cols, theta_pega_poly, theta0_pega_poly
)
test_acc_pega_poly = np.mean(test_preds_pega_poly == test_poly[target_col])
print(f"Pegasos + Poly final test accuracy: {test_acc_pega_poly:.3f}")

Pegasos + Poly best hyperparams => (0.001, 20) CV = 0.9457567516839758
Pegasos + Poly final test accuracy: 0.949

Logistic Regression with PE

Training set cross validation:

acc_logreg_poly = k_fold_cross_validation(
    df=train_poly,
    features=expanded_feature_cols,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr, vl, f, t: logistic_regression_classifier_func(
        tr, vl, f, t,
        lambda_param=best_lam_logreg,
        epochs=best_ep_logreg,
        eta=best_eta_logreg
    )
)

theta_logreg_poly, theta0_logreg_poly = logistic_regression_train(
    train_poly, expanded_feature_cols, target_col,
    lambda_param=best_lam_logreg,
    epochs=best_ep_logreg,
    eta=best_eta_logreg
)
test_preds_logreg_poly = logistic_regression_predict(
    test_poly, expanded_feature_cols, theta_logreg_poly, theta0_logreg_poly
)
test_acc_logreg_poly = np.mean(test_preds_logreg_poly == test_poly[target_col])

print("LogReg + Poly => CV=%.3f, Test=%.3f" % (acc_logreg_poly, test_acc_logreg_poly))
best_hparams_table["LogisticPoly"]["lambda"] = best_lam_logreg
best_hparams_table["LogisticPoly"]["epochs"] = best_ep_logreg
best_hparams_table["LogisticPoly"]["eta"] = best_eta_logreg
best_hparams_table["LogisticPoly"]["cv_accuracy"] = acc_logreg_poly

LogReg + Poly => CV=0.882, Test=0.882

Results

print("\n=== Polynomial-Expanded Feature Set Accuracies ===")
print(f"Perceptron (poly) CV: {acc_perc_poly:.3f},  Test: {test_acc_poly_perc:.3f}")
print(f"Pegasos (poly)    CV: {acc_pega_poly:.3f}, Test: {test_acc_pega_poly:.3f}")
print(f"LogReg (poly)     CV: {acc_logreg_poly:.3f},  Test: {test_acc_logreg_poly:.3f}")

=== Polynomial-Expanded Feature Set Accuracies ===
Perceptron (poly) CV: 0.926,  Test: 0.924
Pegasos (poly)    CV: 0.946, Test: 0.949
LogReg (poly)     CV: 0.882,  Test: 0.882

Compare to Linear (Non-Expanded) Results

We already have our linear results for Perceptron, Pegasos, and Logistic from previous cells:

• accuracy_perc, accuracy_pegasos, accuracy_logreg (5-fold on linear features).

Let's create a small summary:

comparison_poly = pd.DataFrame({
    "Linear": [accuracy_perc, accuracy_pegasos, accuracy_logreg],
    "Polynomial(2)": [acc_perc_poly, acc_pega_poly, acc_logreg_poly]
}, index=["Perceptron", "Pegasos", "Logistic"])

print("Comparison of Linear vs Polynomial(2) feature sets:")
display(comparison_poly)

Comparison of Linear vs Polynomial(2) feature sets:

	Linear	Polynomial(2)
Perceptron	0.585118	0.925513
Pegasos	0.711005	0.945757
Logistic	0.717883	0.882041

Comparison Plot

models = ["Perceptron", "Pegasos", "Logistic"]
linear_accs = comparison_poly["Linear"].values
poly_accs   = comparison_poly["Polynomial(2)"].values
x = np.arange(len(models))
width = 0.35

plt.figure(figsize=(8,4))
plt.bar(x - width/2, linear_accs,  width, label='Linear')
plt.bar(x + width/2, poly_accs,    width, label='Polynomial(2)')
plt.xticks(x, models)
plt.ylabel('CV Accuracy')
plt.title('Linear vs. Polynomial(2) Feature Expansion')
plt.ylim([0, 1])
plt.legend()
plt.tight_layout()
plt.show()

# comparison plot that shows the accuracy of the models on the test set

models = ["Perceptron", "Pegasos", "Logistic"]
linear_accs = [test_acc_perc, test_acc_pega, test_acc_lr]
poly_accs   = [test_acc_poly_perc, test_acc_pega_poly, test_acc_logreg_poly]
x = np.arange(len(models))

plt.figure(figsize=(8,4))
plt.bar(x - width/2, linear_accs,  width, label='Linear')
plt.bar(x + width/2, poly_accs,    width, label='Polynomial(2)')
plt.xticks(x, models)
plt.ylabel('Test Accuracy')
plt.title('Linear vs. Polynomial(2) Feature Expansion')
plt.ylim([0, 1])
plt.legend()
plt.tight_layout()
plt.show()

Weights inspection

theta_perc_poly, theta_0_perc_poly = perceptron_train(
    train_poly, 
    features = expanded_feature_cols, 
    target = target_col,
    epochs = 5
)
print(f"Perceptron (poly) final model trained on entire train_poly. Bias (theta_0)= {theta_0_perc_poly:.3f}")

weights_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_perc_poly,
}).sort_values(by="Weight", ascending=False)

# Show top 10 positive weights and top 10 negative weights
print("\nTop 10 Positive Weights:")
display(weights_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_df.tail(10))

Perceptron (poly) final model trained on entire train_poly. Bias (theta_0)= 28.000

Top 10 Positive Weights:

	Feature	Weight
27	x2*x9	108.472657
10	x8	57.616740
20	x1*x8	46.761335
12	x9	24.724363
6	x5	20.067918
0	x1	17.537309
8	x7	13.122208
44	x3	11.087068
41	x8*x9	7.061366
35	x5*x8	6.677310

Top 10 Negative Weights:

	Feature	Weight
9	x7^2	-3.733855
19	x1*x7	-5.428044
43	x9*x10	-5.533118
5	x4^2	-6.143356
45	x6	-6.267363
13	x9^2	-6.325809
18	x1*x5	-7.307161
17	x1*x4	-12.237412
4	x4	-21.615008
31	x4*x8	-29.930359

theta_pegasos_poly, theta_0_pegasos_poly = pegasos_train(
    train_poly,
    features=expanded_feature_cols,
    target=target_col,
    lambda_param=0.01,
    epochs=5
)
print(f"Pegasos (poly) final model trained on entire train_poly. Bias (theta_0) = {theta_0_pegasos_poly:.3f}")

weights_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_pegasos_poly
}).sort_values(by="Weight", ascending=False)

# Show top 10 positive weights and top 10 negative weights
print("\nTop 10 Positive Weights:")
display(weights_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_df.tail(10))

Pegasos (poly) final model trained on entire train_poly. Bias (theta_0) = 0.622

Top 10 Positive Weights:

	Feature	Weight
27	x2*x9	2.558772
10	x8	1.371412
20	x1*x8	0.801088
6	x5	0.613006
12	x9	0.549907
8	x7	0.470546
0	x1	0.470503
16	x1*x2	0.188492
3	x2^2	0.186295
44	x3	0.146303

Top 10 Negative Weights:

	Feature	Weight
30	x4*x7	-0.046538
24	x2*x5	-0.059779
43	x9*x10	-0.068885
1	x1^2	-0.079423
37	x5*x10	-0.086601
45	x6	-0.118576
5	x4^2	-0.124102
17	x1*x4	-0.215107
4	x4	-0.392167
31	x4*x8	-0.818844

Weights inspection: the three models compared

#Perceptron Weights
weights_perc_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_perc_poly
}).sort_values(by="Weight", ascending=False)

print("=== Perceptron (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_perc_poly:.3f}")

print("\nTop 10 Positive Weights:")
display(weights_perc_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_perc_df.tail(10))


#Pegasos Weights
weights_pega_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_pega_poly
}).sort_values(by="Weight", ascending=False)

print("=== Pegasos (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_pega_poly:.3f}")

print("\nTop 10 Positive Weights:")
display(weights_pega_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_pega_df.tail(10))

#LogReg Weights
weights_logreg_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_logreg_poly
}).sort_values(by="Weight", ascending=False)

print("=== Logistic Regression (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_logreg_poly:.3f}")

print("\nTop 10 Positive Weights:")
display(weights_logreg_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_logreg_df.tail(10))

=== Perceptron (Polynomial) Weights ===
Bias (theta_0) = 2.300

Top 10 Positive Weights:

	Feature	Weight
27	x2*x9	108.472657
10	x8	57.616740
20	x1*x8	46.761335
12	x9	24.724363
6	x5	20.067918
0	x1	17.537309
8	x7	13.122208
44	x3	11.087068
41	x8*x9	7.061366
35	x5*x8	6.677310

Top 10 Negative Weights:

	Feature	Weight
9	x7^2	-3.733855
19	x1*x7	-5.428044
43	x9*x10	-5.533118
5	x4^2	-6.143356
45	x6	-6.267363
13	x9^2	-6.325809
18	x1*x5	-7.307161
17	x1*x4	-12.237412
4	x4	-21.615008
31	x4*x8	-29.930359

=== Pegasos (Polynomial) Weights ===
Bias (theta_0) = 1.222

Top 10 Positive Weights:

	Feature	Weight
27	x2*x9	5.118153
10	x8	2.733301
20	x1*x8	1.707763
12	x9	1.111045
6	x5	1.020537
8	x7	0.792118
0	x1	0.788783
44	x3	0.319826
23	x2*x4	0.226056
14	x10	0.161368

Top 10 Negative Weights:

	Feature	Weight
36	x5*x9	-0.086738
42	x8*x10	-0.115658
37	x5*x10	-0.131599
19	x1*x7	-0.137374
1	x1^2	-0.158771
5	x4^2	-0.277386
17	x1*x4	-0.383653
45	x6	-0.488428
4	x4	-0.727732
31	x4*x8	-1.621416

=== Logistic Regression (Polynomial) Weights ===
Bias (theta_0) = 0.105

Top 10 Positive Weights:

	Feature	Weight
27	x2*x9	0.885225
10	x8	0.562952
20	x1*x8	0.287563
8	x7	0.275661
0	x1	0.238675
6	x5	0.232460
12	x9	0.202625
3	x2^2	0.195755
16	x1*x2	0.169175
13	x9^2	0.164768

Top 10 Negative Weights:

	Feature	Weight
32	x4*x9	-0.023050
33	x4*x10	-0.024077
36	x5*x9	-0.030435
37	x5*x10	-0.036984
24	x2*x5	-0.053120
18	x1*x5	-0.071317
14	x10	-0.081209
45	x6	-0.090833
4	x4	-0.227507
31	x4*x8	-0.245479

Plot

models = ["Perceptron (poly)", "Pegasos (poly)", "Logistic (poly)"]
test_accuracies_poly = [test_acc_poly_perc, test_acc_poly_perc, test_acc_logreg_poly]

plt.figure(figsize=(7,4))
plt.bar(models, test_accuracies_poly, color=["skyblue","salmon","palegreen"])
plt.ylim([0,1])
plt.title("Final Test Accuracies on Polynomial-Expanded Features")
plt.ylabel("Accuracy")
for i, v in enumerate(test_accuracies_poly):
    plt.text(i, v+0.01, f"{v:.3f}", ha='center', fontweight='bold')
plt.tight_layout()
plt.show()

Kernel Methods

Kernelized Perceptron with Gaussian and Polynomial Kernel

k = 5
target_col = "y"

Hyperparameter tuning

sigma_list = [0.1, 0.5, 1.0]
epochs_kperc_gauss = [3, 5]
best_acc_kperc_gauss = 0
best_sigma_kperc = None
best_ep_kperc = None

for sg in sigma_list:
    for ep in epochs_kperc_gauss:
        acc = k_fold_cross_validation(
            df=train_scaled,
            features=final_features,
            target=target_col,
            k=2,  # small because even with k= 5 the result is the same. The result shown below is calculated with k=5.
            random_state=42,
            classifier_func=lambda tr, vl, f, t: kernelized_perceptron_classifier_func(
                tr, vl, f, t,
                kernel_func=gaussian_kernel,
                kernel_params={"sigma": sg},
                epochs=ep
            )
        )
        if acc > best_acc_kperc_gauss:
            best_acc_kperc_gauss = acc
            best_sigma_kperc = sg
            best_ep_kperc = ep

print("\nKernel Perceptron (Gauss) best => sigma=%.3f, epochs=%d, CV=%.3f" % (best_sigma_kperc, best_ep_kperc, best_acc_kperc_gauss))
best_hparams_table["KernelPerceptronGauss"]["sigma"] = best_sigma_kperc
best_hparams_table["KernelPerceptronGauss"]["epochs"] = best_ep_kperc
best_hparams_table["KernelPerceptronGauss"]["cv_accuracy"] = best_acc_kperc_gauss

Kernel Perceptron (Gauss) best => sigma=1.000, epochs=5, CV=0.915

Gaussian Kernel Perceptron

Traning set evaluation

print("=== Kernelized Perceptron (Gaussian Kernel) ===")
sigma_value = 1.0   # Adjust as needed
epochs_gauss = 5    # Adjust as needed

accuracy_gauss = k_fold_cross_validation(
    df=train_scaled,  # or use a smaller subset if runtime is high
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_perceptron_classifier_func(
        tr_fold, vl_fold, f, t,
        kernel_func=gaussian_kernel,
        kernel_params={"sigma": sigma_value},
        epochs=epochs_gauss
    )
)
print(f"Gaussian Kernel: {k}-fold CV Accuracy = {accuracy_gauss:.3f}")

=== Kernelized Perceptron (Gaussian Kernel) ===
Gaussian Kernel: 5-fold CV Accuracy = 0.933

Polinomial Kernel Perceptron

Training Set Evaluation

print("\n=== Kernelized Perceptron (Polynomial Kernel) ===")
degree_value = 2    # e.g., quadratic
c_value = 1.0       # offset in (x^T y + c)^degree
epochs_poly = 5

accuracy_poly = k_fold_cross_validation(
    df=train_scaled, 
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_perceptron_classifier_func(
        tr_fold, vl_fold, f, t,
        kernel_func=polynomial_kernel,
        kernel_params={"degree": degree_value, "c": c_value},
        epochs=epochs_poly
    )
)
print(f"Polynomial Kernel (degree={degree_value}): {k}-fold CV Accuracy = {accuracy_poly:.3f}")

=== Kernelized Perceptron (Polynomial Kernel) ===
Polynomial Kernel (degree=2): 5-fold CV Accuracy = 0.925

Evaluation on Test set

Firstly, the Gaussian Kernel Perceptron

X_train = train_scaled[final_features].values
y_train = train_scaled[target_col].values
X_test = test_scaled[final_features].values
y_test = test_scaled[target_col].values

alpha_gauss = kernelized_perceptron_train(
    X_train, y_train,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_value},
    epochs=epochs_gauss
)

test_preds_gauss = kernelized_perceptron_predict(
    X_train, y_train,
    alpha_gauss,
    X_test,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_value}
)
test_acc_gauss = np.mean(test_preds_gauss == y_test)
print(f"Gaussian Kernel Perceptron Test Accuracy: {test_acc_gauss:.3f}")

Gaussian Kernel Perceptron Test Accuracy: 0.938

Then the polynomial kernel

alpha_poly = kernelized_perceptron_train(
    X_train, y_train,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_value, "c": c_value},
    epochs=epochs_poly
)

test_preds_poly = kernelized_perceptron_predict(
    X_train, y_train,
    alpha_poly,
    X_test,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_value, "c": c_value}
)
test_acc_poly = np.mean(test_preds_poly == y_test)
print(f"Polynomial (d={degree_value}) Kernel Perceptron Test Accuracy: {test_acc_poly:.3f}")

Polynomial (d=2) Kernel Perceptron Test Accuracy: 0.927

Kernelized Pegasos SVM

k = 5
target_col = "y"

Pegasos with Gaussian Kernel

lambda_val = 0.01
epochs_kpega = 5
sigma_val = 1.0

cv_acc_pegasos_gauss = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_pegasos_classifier_func(
        tr_fold, vl_fold,
        features=f, target=t,
        kernel_func=gaussian_kernel,
        kernel_params={"sigma": sigma_val},
        lambda_param=lambda_val,
        epochs=epochs_kpega
    )
)

print(f"Kernelized Pegasos (Gaussian, sigma={sigma_val}) {k}-fold CV accuracy: {cv_acc_pegasos_gauss:.3f}")

Kernelized Pegasos (Gaussian, sigma=1.0) 5-fold CV accuracy: 0.856

Now on the test set

alpha_gauss, T_gauss = kernelized_pegasos_train(
    X_train, y_train,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_val},
    lambda_param=lambda_val,
    epochs=epochs_kpega
)
test_preds_gauss = kernelized_pegasos_predict(
    X_train, y_train,
    alpha_gauss, T_gauss,
    X_test,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_val},
    lambda_param=lambda_val
)
test_acc_kernel_gauss_pega = (test_preds_gauss == y_test).mean()
print(f"Kernelized Pegasos (Gaussian) Test Accuracy: {test_acc_kernel_gauss_pega:.3f}")

Kernelized Pegasos (Gaussian) Test Accuracy: 0.846

Pegasos with Polynomial Kernel

degree_val = 2
c_val = 1.0

cv_acc_pegasos_poly = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_pegasos_classifier_func(
        tr_fold, vl_fold,
        features=f, target=t,
        kernel_func=polynomial_kernel,
        kernel_params={"degree": degree_val, "c": c_val},
        lambda_param=lambda_val,
        epochs=epochs_kpega
    )
)

print(f"Kernelized Pegasos (Polynomial, deg={degree_val}, c={c_val}) {k}-fold CV accuracy: {cv_acc_pegasos_poly:.3f}")

Kernelized Pegasos (Polynomial, deg=2, c=1.0) 5-fold CV accuracy: 0.941

alpha_poly, T_poly = kernelized_pegasos_train(
    X_train, y_train,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_val, "c": c_val},
    lambda_param=lambda_val,
    epochs=epochs_kpega
)
test_preds_poly = kernelized_pegasos_predict(
    X_train, y_train,
    alpha_poly, T_poly,
    X_test,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_val, "c": c_val},
    lambda_param=lambda_val
)
test_acc_kernel_poly_pega = (test_preds_poly == y_test).mean()
print(f"Kernelized Pegasos (Polynomial) Test Accuracy: {test_acc_kernel_poly_pega:.3f}")

Kernelized Pegasos (Polynomial) Test Accuracy: 0.945

Final Data Visualization

final_hparams_df = []
for method, params in best_hparams_table.items():
    row = {"Method": method}
    for k, v in params.items():
        row[k] = v
    final_hparams_df.append(row)

final_hparams_df = pd.DataFrame(final_hparams_df)
print("\n=== Final Hyperparameter Table ===")
display(final_hparams_df)

# (Optional) Save final_hparams_df if you want
# final_hparams_df.to_csv("best_hyperparams.csv", index=False)

=== Final Hyperparameter Table ===

	Method	epochs	eta	cv_accuracy	lambda	sigma
0	Perceptron	10.0	0.1	0.646638	NaN	NaN
1	Pegasos	20.0	NaN	0.724501	0.001	NaN
2	Logistic	10.0	1.0	0.717364	0.100	NaN
3	PerceptronPoly	10.0	0.1	0.925513	NaN	NaN
4	PegasosPoly	20.0	NaN	0.945757	0.001	NaN
5	LogisticPoly	10.0	1.0	0.882041	0.100	NaN
6	KernelPerceptronGauss	5.0	NaN	0.915131	NaN	1.0
7	KernelPerceptronPoly	NaN	NaN	NaN	NaN	NaN
8	KernelPegasosGauss	NaN	NaN	NaN	NaN	NaN
9	KernelPegasosPoly	NaN	NaN	NaN	NaN	NaN

# Example placeholders (DELETE or REPLACE with real computed values):
'''test_acc_perc_linear = 0.80
test_acc_poly_perc = 0.82
test_acc_perc_kernel_gauss = 0.84
test_acc_perc_kernel_poly = 0.83

test_acc_pegasos_linear = 0.81
test_acc_pegasos_poly = 0.83
test_acc_pegasos_kernel_gauss = 0.85
test_acc_pegasos_kernel_poly = 0.84
'''
# If you have logistic regression equivalents, define them too, e.g.:
'''test_acc_logreg_linear = 0.78
test_acc_logreg_poly = 0.80'''

# 1. We'll create a summary DataFrame
final_summary = []

# Perceptron
final_summary.append({
    "Model": "Perceptron",
    "Method": "Linear",
    "Test Accuracy": test_acc_perc
})
final_summary.append({
    "Model": "Perceptron",
    "Method": "Polynomial Expansion",
    "Test Accuracy": test_acc_poly_perc
})
final_summary.append({
    "Model": "Perceptron",
    "Method": "Kernel (Gaussian)",
    "Test Accuracy": test_acc_gauss
})
final_summary.append({
    "Model": "Perceptron",
    "Method": "Kernel (Polynomial)",
    "Test Accuracy": test_acc_poly
})

# Pegasos
final_summary.append({
    "Model": "Pegasos",
    "Method": "Linear",
    "Test Accuracy": test_acc_pega
})
final_summary.append({
    "Model": "Pegasos",
    "Method": "Polynomial Expansion",
    "Test Accuracy": test_acc_pega_poly
})
final_summary.append({
    "Model": "Pegasos",
    "Method": "Kernel (Gaussian)",
    "Test Accuracy": test_acc_kernel_gauss_pega
})
final_summary.append({
    "Model": "Pegasos",
    "Method": "Kernel (Polynomial)",
    "Test Accuracy": test_acc_kernel_poly_pega
})

# (Optional) Logistic
final_summary.append({
    "Model": "Logistic",
    "Method": "Linear",
    "Test Accuracy": test_acc_lr
})
final_summary.append({
    "Model": "Logistic",
    "Method": "Polynomial Expansion",
    "Test Accuracy": test_acc_logreg_poly
})

summary_df = pd.DataFrame(final_summary)
summary_df = summary_df.sort_values(by=["Model", "Method"]).reset_index(drop=True)
print("Final Test Accuracy Summary:")
display(summary_df)


# %%
# Barplot of final test accuracies by Model and Method
# Prepare data
models = summary_df["Model"].unique()
methods = summary_df["Method"].unique()

# Pivot the DataFrame for plotting
pivot_df = summary_df.pivot(index="Model", columns="Method", values="Test Accuracy")
display(pivot_df)

# Plot grouped bar chart
pivot_df.plot(kind="bar", figsize=(10,6))
plt.ylim([0, 1])
plt.title("Final Test Accuracies by Model and Method")
plt.ylabel("Accuracy")
plt.legend(title="Method", bbox_to_anchor=(1.0, 1.0))
plt.tight_layout()
plt.show()

# Single bar chart with all methods (Model x Method)
plt.figure(figsize=(8,6))
x_vals = range(len(summary_df))
acc_vals = summary_df["Test Accuracy"].values
labels = [f"{row.Model}\n{row.Method}" for idx, row in summary_df.iterrows()]

plt.bar(x_vals, acc_vals, color="skyblue")
for i, v in enumerate(acc_vals):
    plt.text(i, v+0.01, f"{v:.3f}", ha='center', fontweight='bold')
plt.ylim([0,1])
plt.xticks(x_vals, labels, rotation=45, ha="right")
plt.title("Final Test Accuracies (All Experiments)")
plt.tight_layout()
plt.show()

Final Test Accuracy Summary:

	Model	Method	Test Accuracy
0	Logistic	Linear	0.733950
1	Logistic	Polynomial Expansion	0.882383
2	Pegasos	Kernel (Gaussian)	0.845917
3	Pegasos	Kernel (Polynomial)	0.945044
4	Pegasos	Linear	0.732922
5	Pegasos	Polynomial Expansion	0.949153
6	Perceptron	Kernel (Gaussian)	0.937853
7	Perceptron	Kernel (Polynomial)	0.927067
8	Perceptron	Linear	0.670262
9	Perceptron	Polynomial Expansion	0.923986

Method	Kernel (Gaussian)	Kernel (Polynomial)	Linear	Polynomial Expansion
Model
Logistic	NaN	NaN	0.733950	0.882383
Pegasos	0.845917	0.945044	0.732922	0.949153
Perceptron	0.937853	0.927067	0.670262	0.923986

save